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1 .  List  of  Appendixes,  Illustrations  and  Tables  (if  applicable) 


Figure  1  Illustration  of  the  proposed  human-UAV  cooperative  decision  system,  where  UAVs  are  controlled  actively  (through  a 
thought-based  commend)  and  passively  (through  sensing  the  cognitive  states)  by  soldiers’  thoughts  detected  by  EEG  sensors 
placed  in  their  helmet. 

Figure  2  The  acquired  Active  Two  256-channel  Biosemi  systems  and  the  EEG  lab  established  at  UTSA 
Figure  3  Block  diagram  of  the  proposed  SSVEP-based  brain-controlled  UAV  system. 

Figure  4  Illustration  in  the  SSVEP  BCI  system  that  implements  6  commends.  (A)  the  arrangement  of  the  flickering  and  the 
frequencies.  (B)  Illustration  of  a  participant  looking  at  flickering  visual  stimulus  in  the  real  system. 

Figure  5  The  locations  of  the  12  electrodes  used  in  this  system. 

Figure  6  Power  spectrum  for  a  10Hz  (left)  and  15Hz  (right)  trial.  Different  curves  correspond  to  different  channels.  Significant 
peaks  at  10Hz  (left)  and  15Hz  (right)  and  their  respective  harmonics  can  be  clearly  seen. 

Figure  7  Average  error  rates  of  the  experiment  involving  40  participants,  each  conducted  3  trials. 

Figure  8  Screenshots  of  real  demonstration  of  brain-control  quad-copter  flight  missions. 

Figure  9  .  A  The  general  DSN  architecture  with  three  stacking  modules  (dotted  blue  boxes).  Note  input  (V)  and  output  (Yn-1) 
are  concatenated  to  form  the  input  the  2nd  and  third  module.  B.  The  detailed  architecture  of  a  DSN  module,  which  is  a 
simplified  MLP  that  includes  a  single  layer  RBM  followed  by  a  linear  classification  layer.  C.  The  detailed  architecture  of  the 
proposed  DSNTF  module,  which  includes  a  set  of  weights  (with  superscript  “A”)  trained  from  other  subjects  and  another  set 
(with  superscript  “O”)  trained  specifically  for  the  SOI. 


2.  Statement  of  the  problem  studied 

One  key  objective  of  the  project  in  this  reporting  period  is  to  acquire  high  performance  EEG  Biosemi  systems.  In  addition  to  our 
effort  in  acquiring  the  EEG  systems,  we  have  also  been  working  on  two  proposed  projects  in  the  area  of  brain  computer 
interface  (BCI). 

2.1  Acquisition  of  high  performance  EEG  systems 

The  goal  of  this  project  is  to  acquire  high  performance  Electroencephalogram  (EEG)  systems  that  enable  real-time 
measurement  of  human  brain  activities  at  high  spatial-temporal  resolution  in  both  laboratory  and  real-life  environments.  The 
specific  aims  are  to  enable  1)  multi-subject  capability;  2)  real-time  measurement  of  brain  wave  with  highest  possible  resolution 
at  a  laboratory  environment;  3)  real-time  measurement  of  brain  activity  in  real  life  environments.  In  addition,  we  also  intend  to 
acquire  additional  EEG  devices  suitable  for  classroom  and  research-based  educational  purpose.  We  propose  to  acquire  1)  the 
most  advanced  Biosemi  Active  Two  EEG  device  to  enable  real-time  measurement  of  brain  wave  with  highest  possible 
resolution  at  a  laboratory  environment,  and  2)  dry  electrode,  wearable  wireless  EEG  caps  to  measure  brain  activities  in  real  life 
environment  and  for  educational  use.  For  this  reporting  period,  we  focused  on  the  acquisition  of  Biosemi  Active  Two  EEG 
devices. 

2.2  Controlling  a  Simulated  Quad-copter  using  a  SSVEP-based  Brain-Computer  Interface 

Our  long  term  goal  is  to  develop  and  demonstrate  core  technologies  to  control  cooperative  small  unmanned  aerial  vehicles 
(SUAVs)  to  complete  simple  missions  using  brain  waves  of  a  soldier  (Fig.  1 ).  There  has  been  substantial  amount  of  technical 
advances  made  over  the  past  two  decades  in  the  area  of  autonomous  unmanned  vehicles.  The  miniaturization  of 
computational,  sensory,  and  communicational  components  have  accelerated  the  pace  of  the  recent  advancement.  Most  of 
previous  works  dealt  with  a  variety  of  user  interface  issues  for  control  stations.  What  has  not  been  studied  in  depth  is  ways  for 
soldiers  to  interact  directly  with  SUAVs.  However,  successful  SUAV  mission  executions  rely  heavily  on  the  ability  to  sense 
critical  information  of  a  mission  situation  from  different  complimentary  sensor  resources.  While  these  sensors  have  proven  to 
provide  information  that  can  lead  to  improved  decision  making,  soldiers’  comprehension  of  complex  environments,  including  the 
collective  sensor  information,  and  their  cognitive  decisions  are  often  the  most  valuable,  enabling  effective  and  efficient 
execution  of  complex  SUAV  missions.  Integrating  soldiers  into  the  cooperative  decision  making  of  SUAVs  is  imperative. 
Furthermore,  due  to  pervasive  threats  from  enemies  that  employ  unconventional  tactics,  it  would  be  highly  desirable  to  have  an 
integrated,  human-SUAV,  sociotechnical  system,  where  SUAVs  possess  the  ability  to  sense  the  potential  risks  from  soldiers’ 
high  level  cognitive  commands  and  act  accordingly  to  avert  the  risks. 


The  ultimate  goal  of  this  project  is  to  design  a  BCI  system  to  control  a  group  of  SUAVs  to  perform  a  simple  cooperative  mission, 
such  as  a  simple  search  mission  and  an  Intelligence  Collection,  Surveillance,  and  Reconnaissance  (ISR)  mission  by  only  using 
brain  waves  of  a  human.  The  goal  for  this  reporting  period  is  to  design  a  BCI  system  based  on  the  Steady-State  visually 
Evoked  Potentials  (SSVEP)  method  to  interface  with  a  Quad-copter  simulator  to  control  the  takeoff  (which  also  hovers)  and 
landing  using  brain  wave. 

2.3  A  deep  neural-network  system  for  cross-subject  and  cross-experiment  prediction  of  image  rapid  serial  visual  presentation 
events 

We  propose  to  study  in  the  project  a  particular  BCI  system  that  uses  brain  signals  to  detect  rare  target  images  within  a  large 
collection  of  non-target  images.  Such  BCI-based  systems  can  significantly  increase  the  speed  of  target  image  identification  in 
large  image  databases  over  manual  procedures.  Data  collection  for  training  such  BCI  systems  is  commonly  carried  out  under  a 
rapid  serial  visual  presentation  (RSVP)  paradigm,  where  test  subjects  are  asked  to  identify  a  target  image  from  a  continuous 
burst  of  image  clips  presented  at  a  high  rate.  The  EEC  recordings  or  their  power  spectrums  within  300-1 000ms  immediately 
following  the  onset  of  target  and  non-target  images  are  collected  and  a  classifier  capable  of  predicting  the  presence  of  target 
images  based  on  EEC  responses  is  trained  using  this  data. 

The  prediction  of  target  images  using  the  trained  classifier  can  be  categorized  as  a  within-subject,  cross-subject,  or  cross¬ 
experiment  task,  depending  upon,  if  the  prediction  data  come  from  the  same  subject,  different  subjects,  or  different  RSVP 
experiment  as  the  training  data  source,  respectively.  So  far,  a  number  of  classifiers  including  logistic  regression,  linear 
discriminant  analysis  (LDA),  support  vector  machines  (SVM)  and  hierarchical  discriminant  component  analysis  (HDCA)  have 
been  proposed  in  the  literature.  Although  some  of  these  classifiers  have  been  shown  to  achieve  high  accuracy  for  within- 
subject  prediction,  all  of  them  suffer  significant  performance  degradation  when  applied  for  a  cross-subject  or  cross-experiment 
task.  This  degradation  can  be  largely  attributed  to  the  increased  differences  in  brain  activities  across  subjects  and  experiments. 
As  demonstrated  in  the  literature,  the  target  event  related  potential  (ERP)  can  change  in  both  magnitude  and  timing  with  subject 
and  experiment.  To  improve  the  prediction  performance,  new  methods  that  can  extract  robust  EEC  features  across  subjects 
and  experiments  and  can  adapt  to  the  changes  of  subjects  and  experiments  need  to  be  developed.  Moreover,  It  would  be  of 
significant  practical  interest  to  develop  a  prediction  system  that  can  integrate  existing  data  from  other  subjects  with  that  from  the 
subject  of  interest  to  improve  the  prediction  performance.  However,  given  the  potential  differences  in  brain  activities  across 
subjects  and  experiments,  it  is  unclear  if  and  how  data  from  other  subjects/experiments  can  be  transferred  to  improve  the  target 
prediction.  In  this  project,  we  investigated  deep  learning  (DL)  classifiers  for  cross-subject  and  cross-experiment  prediction  of 
RSVP  target  events  and  we  also  proposed  a  novel  deep  transfer  learning  algorithms  that  can  integrate  data  from  other  subjects 
for  improved  target  prediction.  Deep  learning  is  a  new  family  of  learning  methods  with  a  multiple-layered  architecture  that  have 
been  shown  to  offer  superb  representation  of  complex  data.  DL  has  gained  great  interest  in  recent  years  due  to  its  ability  to 
outperform  other  classification  methods  in  several  machine  learning  competitions  and  in  a  variety  of  applications  including 
image  classification  and  speech  recognition.  Its  application  in  EEG-based  BCI  data  analysis  is  still  at  a  very  early  stage.  We 
have  previously  shown  on  a  small  dataset  that  deep  learning  algorithms  achieved  about  5%  performance  gain  over  other  exiting 
algorithms  for  within-subject  RSVP  target  prediction.  However,  more  studies  need  to  be  conducted  to  confirm  the  improvement 
of  DL  classifier  for  within-subject  prediction  based  on  relative  small  training  data  because  the  superior  performance  of  DL  in 
other  applications  is  almost  always  achieved  as  a  result  of  enormous  training  data  (e.g.  millions  of  training  images). 

Furthermore,  if  and  how  DL  can  improve  cross-subject  or  cross-experiment  prediction  based  on  limited  training  samples  has  not 
been  explored. 

3.  Summary  of  the  most  Important  results 

We  first  summarize  in  the  following  important  results  for  each  of  the  3  tasks  stated  in  section  “Statement  of  the  problem 
studied”.  We  will  then  discuss  the  outcome  of  outreach  and  educational  efforts. 


3.1  Acquisition  of  high  performance  EEG  systems 

We  have  purchased  two  (2)  256-channel  Biosemi  Active  Two  EEG  devices.  This  biopotential  measurement  system  has  256 
active  electrodes  and  24-blt  resolution.  It  is  the  most  advanced  Biosemi  EEG  system  that  offers  to  measure  brain  activity  at  the 
highest  possible  spatial  and  temporal  resolution.  Each  system  includes  an  ActiveTwo  Base  System  that  includes  A/D  Box,  2 
battery  Units,  battery  Charger,  USB  2.0  Interface,  Interface  Cable,  and  acquisition  Software,  an  amplifier/converter  module,  an 
auxiliary  8-channel  input  box,  256  sensors,  and  3  head  cap  with  different  size.  In  addition,  we  have  acquired  one  64-channel, 
one  32-channel,  and  two  16-channel  Cognlonics  dry  sensor  headsets.  We  have  also  set  up  a  110  ft2  EEG  room  for  storing  the 
EEG  systems  and  conducting  experiments  (Fig.  2). 


3.2  Controlling  a  Simulated  Quad-copter  using  a  SSVEP-based  Brain-Computer  Interface 


The  main  objective  of  this  project  is  to  control  a  simple  Unmanned  Aerial  Vehicle  (UAV)  using  the  brain  activity,  specifically 
using  a  Brain-Computer  Interface  (BCI)  system.  The  overall  block  diagram  of  the  designed  system  can  be  found  in  Fig.  3.  The 
main  BCI  element  Is  a  Steady-State  Evoked  Potentials  (SSVEP)  system  that  functions  to  induce  a  specific  pattern  in  brain 
signals  associated  with  a  particular  command  (e.g.  turning  left).  The  command  associated  signals  are  detected  by  the 
Canonical  Correlation  Analysis  (CCA)  algorithm  and  sent  to  the  control  unit  or  Robot  Operating  System  (ROS)  of  a  quad-copter 
(the  UAV),  which  will  execute  the  specific  command. 

3.2.1  Introduction  to  SSVEP 

Several  approaches  have  been  developed  for  BCI  and  they  make  It  possible  to  control  an  electronic  device  using  the  subject's 
neural  activity.  The  approach  chosen  for  this  project  is  called  Steady-State  Evoked  Potentials  (SSVEP).  In  brief,  the  main 
principle  behind  this  approach  is  that  when  the  subject  observes  a  visual  stimulus  that  flickers  at  a  certain  frequency  f,  the 
neural  activity  generated  in  the  visual  cortex  registers  (among  other  activities)  a  sinusoidal-like  signal  whose  frequency  Is  also  f. 
This  fact  makes  it  possible  to  design  a  BCI  system  that  codes  stimulus  frequencies  as  commands  to  be  sent  to  the  output 
device.  By  accurately  recognizing  the  frequency  of  the  visual  stimulus  the  subject  is  looking  at,  the  coded  command  can  be 
detected. 

SSVEP  belongs  to  the  REACTIVE  type  of  BCI,  which  means  the  subject  Is  not  concentrated  in  giving  direct  orders  with  his 
mind.  The  design  of  an  SSVEP  experiment  is  usually  quick,  simple  and  inexpensive.  SSVEP  is  a  very  active  research  topic 
within  the  BCI  field  and  several  detections  algorithms  have  been  tested  and  diverse  kind  of  applications  have  been  built. 
Important  factors  that  need  to  be  carefully  considered  when  designing  a  SSVEP  system  are  listed  in  the  following: 

•  The  visual  stimulus:  shape,  size,  brightness,  color,  etc.  are  factors  that  influence  the  SSVEP  performance;  using  flickering 
images  as  source  of  stimulus  has  been  documented. 

•  Focal  distance,  concentration  level,  fatigue  and  other  mental  and  physical  states  also  affect  the  SSVEP  performance. 

•  The  visual  stimulus  are  usually  presented  using  arrays  of  LEDs,  TVs  or  computer  monitor.  When  using  TV  or  computer 
monitors,  the  refresh  rate  of  the  screen  has  to  be  taken  in  consideration  to  correctly  reproduce  the  flickering  frequencies. 

•  Frequencies  ranging  as  low  as  6Hz  to  high  as  90Hz  have  been  successfully  used  for  SSVEP. 

•  Minimum  training  is  required  for  an  SSVEP  experiment. 

3.2.2  The  proposed  SSVEP  experiment 

The  objective  of  the  experiment  is  to  capture  the  sine-like  signal  using  EEG,  which  is  elicited  when  the  subject  watches  the 
SSVEP  visual  stimulus.  An  illustration  of  our  SSEVP  experiment  Is  shown  in  Fig.  4.  The  detailed  design  of  the  experiment  can 
be  summarized  as  follows: 

•  A  24  inches  computer  monitor  is  placed  in  front  of  the  subject,  approximately  30  -  50  cm  focal  distance.  The  visual  stimulus 
corresponding  to  two  flickering  squares  are  located  on  the  left  and  right  borders  of  the  screen  at  a  central  high.  Each  square  is 
approximately  5  cm  length  per  side. 

•  Participant  is  wearing  an  EEG  cap  with  12  electrodes  (Fig.  5).  Several  different  arrangements  of  sensors  on  the  scalp  have 
been  used  in  the  literature.  In  our  case,  the  12  channels  are  spread  throughout  several  locations  around  the  scalp  in  a 
symmetric  fashion,  emphasizing  particularly  the  area  of  visual  cortex.  The  sampling  rate  of  the  system  is  512Hz. 

•  Visual  stimulus  has  six  flickering  with  6.0Hz,  7.5Hz,  8.57Hz,  10Hz,  10.9Hz  and  13.3  Hz  (Fig.  4-(A)). 

•  Participant  starts  in  neutral  position,  looking  at  the  fixation  mark.  An  arrow  indicator  displays  for  7  seconds,  directing 
participant  to  a  specific  flicker.  After  7  seconds,  the  arrow  disappears  and  the  fixation  mark  appears  for  7  seconds.  This  arrow 
rotates  clockwise  with  the  7  second  intervals.  This  rotation  occurs  3  times  during  a  trial  for  a  total  of  18  views  of  the  various 
flickers.  After  viewing  18  flickers  the  trial  finishes  and  the  flickers  disappears,  participant  has  40  seconds  to  rest  before  the  next 
cycle  begins 

•  Presentation  program  was  implemented  in  MATLAB  using  the  Psychophysics  Toolbox  (PTB)  for  the  low-level 
communications  with  the  computer  monitor  through  OpenGL. 

•  The  lab-streaming  layer  (LSL)  library  was  used  to  stream  the  EEG  recordings  from  BioSemi  hardware  and  combined  with 
the  SSVEP  presentation  data  (event  labels,  timing,  etc.). 


3.2.3  EEG  acquisition  and  preprocessing 

EEG  signals  were  recorded  using  a  Blosemi  Active  Two  System.  As  mentioned  before,  12  electrodes  were  used  on  the 
standard  adult-size  256  electrodes  cap.  The  SSVEP  presentation  program  and  the  EEG  recording  were  performed  on  the  same 
computer.  For  the  experiment,  EEG  data  were  recorded  using  the  built-in  Labview-based  Biosemi  acquisition  software. 

The  data  pre-processing  was  conducted  using  MATLAB  and  the  EEGLAB  toolbox.  After  signal  was  correctly  loaded  and 
trimmed  (initial  seconds  of  PTB  initialization),  the  following  operations  were  carried  out  for  pre-processing: 

•  Downsampling  to  128  Hz 

•  FIR  band-pass  filter  with  frequencies  1  Hz  to  60Hz 

•  Averaged  re-referencIng 


Baseline  removal 


After  these  steps  of  preprocessing,  the  power  spectrum  was  generated  to  verify  if  there  was  frequency  spikes  at  the  stimulus 
frequency  and  their  harmonics  as  suggested  by  the  SSVEP  theory.  An  example  of  a  1 0Hz  and  1 5Hz  frequency  response  for  a 
single  trial  can  be  seen  in  Fig.  6. 

3.2.4  Detection  of  SSVEP  frequencies  with  Canonical  Correlation  Analysis 

We  investigated  the  Canonical  Correlation  Analysis  (CCA)  algorithms  for  automatic  detection  of  the  SSVEP  frequencies  and 
evaluated  its  performance  for  our  experiment.  In  a  nutshell,  the  CCA  algorithm  performs  correlations  between  a  sinusoidal 
signal  and  its  harmonics  with  the  predefined  frequency  (e.g.  10Hz  or  15Hz)  and  EEC  recording  from  different  channels  and  then 
finds  a  linear  transformation  that  maximizes  the  combination  of  the  correlations  or  the  CCA  correlation.  The  frequency  that 
associated  with  the  largest  CCA  correlation  is  detected  as  the  stimulus  frequency. 

3.2.5  Investigate  the  performance  of  the  SSVEP  system. 

To  evaluate  the  performance  of  the  designed  SSVEP  system,  we  conducted  experiments  of  40  subjects,  each  performed  3 
trials  that  last  for  about  1 5  seconds.  Specifically,  the  participant  was  asked  to  sit  still  in  trial  1  and  2  but  allowed  to  talk  and 
move  in  trial  3.  Data  were  spitted  into  a  3-second  length  epochs,  where  each  epoch  contains  only  a  single  stimulus.  A  total  of 
192  epochs  were  obtained  from  all  the  recorded  data  for  each  participant  in  each  trial.  CCA  was  applied  to  each  epoch  and  the 
detection  frequency  was  compared  with  the  true  frequency  of  the  flickering.  An  error  is  record  if  the  predicted  and  the  true 
frequencies  did  not  agree.  We  obtain  an  average  error  rate  of  0.0996,  0.1466  and  0.2936  for  trial  1, 2  and  3,  respective  (Fig  7). 
We  observed  that  the  average  error  rate  is  low  in  trial  1  &2  but  increases  significantly  in  trial  3.  We  performed  T-test  to  test  the 
differences  between  the  trials.  The  test  determines  that  there  is  no  significant  differences  between  the  error  rate  of  trial  1  &2 
(p<0.05)  but  shows  a  significant  difference  between  the  error  rates  between  trail  1  &3  or  2  &  3.  Since  the  difference  between 
trial  1  (or  2)  &  3  is  the  allowed  movement  of  the  participant,  we  conclude  that  movement  during  SSVEP  experiment  significantly 
impairs  the  detection  performance. 

3.2.6  Execution  of  the  brain-controlled  quadcopter  flight  operations 

As  the  final  step,  we  connected  the  SSVEP  BCI  system  with  the  real  quadcopter  and  performed  two  types  of  flight  mission.  The 
first  mission  involves  one  quadcopter  and  six  low  level  commends  are  executed,  which  includes  “Move  Forward”,  “Move 
Backward”,  “Move  Right”,  “Move  Left”,  “Move  Up”,  and  “Move  Down”.  Using  these  commends,  we  successfully  demonstrated 
the  direct  execution  of  these  commends  and  the  square  formation.  The  second  mission  includes  controlling  two  drones  to 
execution  high  level  commands  including  “Scatter”  (two  drones  move  away  from  each  other)  and  “Gather”  (two  drones  move 
closer  to  each  other),  which  we  were  able  to  successfully  implemented  as  well. 


3.3  Deep  learning  methods  for  cross-subject  and  cross-experiment  prediction  of 
image  rapid  serial  visual  presentation  events 

We  investigated  the  performance  of  Deep  Stacking  Network  (DSN)  and  deep  neural  network  (DNN),  two  most  popular  deep 
learning  algorithms,  for  cross-subject  and  cross-experiment  prediction  of  RSVP  targets.  In  addition,  we  also  proposed  a  new 
deep  transfer  learning  algorithm,  where  we  augment  the  training  data  of  a  subject  of  interest  (SOI)  with  data  from  other 
subjects. 

3.3.1  A  new  transfer  learning  deep  stacking  network  algorithm 

The  goal  of  transfer  learning  is  to  augment  the  training  data  of  a  subject  of  interest  (SOI)  with  data  from  other  subjects  from 
either  the  same  of  different  experiment.  However,  simply  augmenting  data  from  other  subject  might  not  help  improve  the 
performance  for  SOI.  Especially,  the  training  data  from  other  subject  can  be  of  much  larger  size  than  that  from  SOI.  In  this  case, 
the  discriminant  features  of  SOI  can  be  significantly  skewed  by  virtue  of  the  enormous  data  size  from  other  subjects.  This  could 
be  beneficial  when  data  from  other  subject  carries  more  discriminant  information  but  it  could  equally  be  detrimental  when  target 
event  related  brain  activities  from  other  subjects  are  not  as  discriminant.  Therefore,  it  is  desiratsle  to  develop  an  algorithm  that 
can  transfer  only  the  useful,  discriminant  information  of  other  subject  to  SOI.  We  propose  a  novel  DSN  architecture  for  transfer 
learning  or  DSNTF  (Fig.  9  ).  The  uniqueness  of  DSNTF  is  that  it  is  essentially  an  augmentation  of  a  SOI  DSN  with  a  cross¬ 
subject  DSN.  To  train  this  DSNTF,  we  first  train  a  cross-subject  DSN  suing  data  from  other  subjects.  Then,  the  SOI  DSN  is 
trained  using  only  data  from  SOI.  However,  before  training,  the  hidden  units  in  each  module  are  expanded  by  including  those 
from  the  trained  cross-subject  DSN,  where  the  corresponding  RBM  weights  for  the  cross-subject  hidden  units  are  also  retained. 
The  training  thus  involves  learning  the  weights  for  the  SOI  hidden  units  as  well  as  updating  the  weights  for  cross-subject  hidden 
units  using  the  subject-specific  data.  Because  the  training  of  each  DSNTF  module  should  maximize  the  classification 
performance,  weight  updating  for  cross-subject  hidden  units  functions  to  transfer  the  discriminant  information  learned  from  other 


subjects  about  the  target  event  to  the  individualized  DSN  designed  for  the  subject  of  interest.  As  a  result,  this  new  architecture 
integrates  subject-specific  knowledge  and  cross-subjects  knowledge  about  the  target  event  to  produce  a  prediction  that  not  only 
emphasizes  individual  characteristics  in  brain  activity  but  also  reflects  the  subject-wide  consensus  that  defines  the  global  brain 
activity  underlying  the  target  event.  Because  the  cross-subject  hidden  units  bring  the  global,  potentially  more  robust  features 
into  the  classification,  this  new  architecture  is  likely  Immune  to  data  noise  in  outlier  samples,  thus  making  the  individualized 
prediction  more  robust.  At  the  same  time,  when  cross-subject  hidden  units  are  less  discriminant,  the  updating  process  should 
function  to  discount  these  cross-subject  hidden  units,  thus  maintaining  a  performance  close  to  that  of  the  within-subject. 

3.3.2  DL  algorithms  showed  significant  improvement  over  existing  algorithms. 

We  evaluated  the  performance  of  DSN  and  DNN  for  within-subject  target  classification  for  two  RSVP  datasets  ,  the  static 
motion  and  the  Cognitive  Technology  Threat  Warning  System  (CT2WS).  The  static  motion  experiment  include  presentation  of 
images  of  enemy  soldiers/combatants  (target)  versus  the  background  image  (non-target).  In  the  CT2WS  experiment,  target 
images  include  moving  people  and  vehicle  animations,  whereas  the  non-targets  are  other  types  of  animation.  For  both 
experiments,  the  images  were  presented  at  2Hz  (one  image  presented  every  500ms)  and  brain  signals  were  recorded  with  64- 
channel  Biosemi  EEG  systems  at  a  sampling  rate  of  512  Hz.  There  were  a  total  of  16  and  15  subjects  in  the  static  motion  and 
CT2WS  experiment,  respectively.  For  both  experiments,  DNN  and  DSN  both  had  very  similar  Area  Under  the  ROC,  or  Az 
performance  and  DNN  reported  an  Az  score  of  94.5%  for  the  static  motion  RSVP  experiment.  When  compared  with  the  state- 
of-the-art  HDCA  method,  the  two  DL  algorithms  reported  >  6%  improvements  in  Az  scores  for  both  experiments.  Taken 
together,  this  result  demonstrated  good  performance  and  a  clear  improvement  achieved  by  DL  algorithms  for  within-subject 
target  prediction. 

3.3.3  DL  classifiers  are  more  robust  for  cross-subject  and  cross-experiment  predictions. 

We  investigated  how  robust  DL  algorithms  when  used  to  predict  the  target  event  for  a  test  subject  different  from  those  for 
training.  We  first  evaluated  their  cross-subjects  performance,  where  the  test  subject  and  training  subjects  are  from  the  same 
experiment.  As  expected,  both  HDCA  and  LDA  deteriorated  significantly  relative  to  their  within-subject  performance,  where  the 
Az  scores  of  HDCA  dropped  for  more  than  10%  from  0.87  to  0.74  for  the  static  motion  and  0.8  to  0.7  for  the  CT2WS.  In 
contrast,  both  DNN  and  DSN  demonstrated  a  much  robust  performance,  suffered  only  4.7%  and  2.1%  performance  degradation 
for  the  static  motion  and  0.8  to  0.7  for  the  CT2WS,  respectively. 

We  then  examined  if  this  more  robust  performance  can  still  be  observed  for  cross-experiment  prediction,  which  is  a  more 
difficult  task  than  cross-subject  prediction  because  the  test  subject  Is  from  an  experiment  different  from  the  training  subject.  For 
this  test,  we  trained  the  classifiers  using  all  subject  data  from  the  static  motion  RSVP  experiment  and  then  evaluated  their 
performance  for  each  subjects  from  the  CT2WS  experiment.  As  expected,  the  performance  of  all  tested  classifiers  including 
DNN  and  DSN  degraded  more  than  cross-subject  performance.  However,  the  two  DL  algorithms  registered  a  less  degradation 
in  performance  than  either  HDCA  or  LDA  (-12%  for  DL  from  0.86  to  0.76  vs.  -16%  for  HDCA  from  0.8  to  0.67  and  28%  for  LDA 
from  0.74  to  0.58).  Taken  together,  the  results  indicated  that  DL  algorithms  are  more  robust  for  both  cross-subject  and  cross¬ 
experiment  predictions. 

3.3.4  The  proposed  transfer  learning  DSN  improves  over  the  within-subject  prediction  performance 

We  investigated  if  the  proposed  transfer  learning  schemes  can  improve  the  within-subject  performance  by  integrating  additional 
data  from  other  subjects  either  from  the  same  experiment  or  different  experiment.  For  the  within-experlment  transfer  learning, 
similar  procedure  as  the  within-subject  test  was  followed  except  that  the  within-subject  training  data  is  augmented  also  by  data 
from  other  subjects  of  the  same  experiment.  Surprisingly,  directly  augmenting  the  training  data  from  other  subjects  appeared  to 
harm  rather  than  improve  the  within-subject  performance.  This  result  suggests  that  there  exist  differences  in  the  learned 
discriminant  EEG  features  for  different  subjects  and  many  discriminant  EEG  features  for  other  subjects  would  not  be 
discriminant  for  the  subject  of  interest.  As  a  result,  including  data  from  other  subjects  might  add  more  non-discriminant  features 
and  subsequently  degrade  the  performance.  However,  the  fact  that  DNN  with  data  augmentation  (DNNDA)  and  DSN  with  data 
augmentation  (DSNDA)  only  suffered  a  slight  drop  in  performance  suggests  once  again  that  the  discriminant  features  learnt  by 
the  DL  algorithms  are  more  robust  cross  subjects.  In  contract,  the  performance  of  HDCA  and  LDA  deteriorated  much  closer  to 
their  corresponding  cross-subject  performance.  This  observation  could  suggest  that  the  discriminant  EEG  features  learnt  by 
HDCA  and  LDA  cross  subjects  are  potentially  very  different.  The  learnt  discriminant  features  became  similar  to  those  by  cross¬ 
subject  training,  hence  the  performance,  because  after  data  augmentation  there  were  much  more  data  samples  from  cross 
subjects  than  subject  of  interest. 

On  the  contrary,  we  noticed  that  the  proposed  transfer  learning  DSN  did  achieved  improvement  over  the  within-  subject  DSN. 
This  result  substantiated  our  original  motivation  for  proposing  the  transfer  learning  architecture.  Indeed,  this  TF  architecture 
enables  to  transfer  these  cross-subjects  discriminant  features  that  are  only  beneficial  to  the  subject  of  interest.  The  result  that 
DSNTF  achieved  almost  the  same  performance  as  within-subject  DSN  for  the  CT2WS  experiment  also  demonstrated  that  when 
the  cross-subjects  data  appear  very  different,  this  architecture  seems  to  allow  potentially  no  features  transferred. 


We  further  validated  this  result  about  transfer  learning  by  augmenting  data  of  subjects  from  different  experiments.  HDCA  and 


LDA  suffered  more  significant  deterioration  because  of  the  impact  of  additional  cross-experimental  differences.  As  expected, 
DSNDA  and  DNNDAonly  have  small  drops  in  performance.  Overall,  DSNTF  still  reported  the  best  performance  which  is  only 
1%  lower  than  that  of  within-subject  DSN.  This  result  confirmed  that  the  conceived  advantages  of  the  proposed  transfer  learning 
DSN. 

3.4  Education  and  outreach  activities 

Overall,  we  have  involved  2  PhD  students,  one  MS  student,  and  2  undergraduate  students  in  conducting  research  that  are 
related  to  this  project. 

We  hosted  Kristine  Diaz,  a  female,  Hispanic  undergraduate  student  from  the  Alamo  College  to  implement  a  SSVEP  based  BCI 
system  from  July-Aug,  2014.  Kristine  Diaz  was  selected  through  the  Alamo  College  Summer  Research  Program,  a  National 
Science  Foundation  funded  program.  Ms.  Diaz  has  participated  in  the  SSVEP  experiments  both  as  a  test  subject  and  in  helping 
out  with  the  experiments.  She  has  also  involved  in  data  collection,  preprocessing,  and  analysis.  She  has  reported  her  research 
experience  and  results  from  this  project  in  a  poster  presented  the  final  research  conference  of  the  program. 

We  have  also  recruited  Jonathan  David  Upchurch  and  Garrett  Hall,  two  UTSA  electrical  engineering  undergraduate  students 
and  Air  Force  veterans,  to  work  on  the  brain-controlled  UAV  project.  Both  of  them  have  been  helping  conducting  SSVEP 
experiments  and  research  the  dry  sensor  EEG  systems. 

We  have  also  mentored  an  EE  senior  design  team  to  implement  a  BCI  system  that  monitors  specific  areas  in  the  frontal  lobe 
that  deal  with  concentration  and  focus.  The  senior  design  project  won  the  2nd  place  in  the  UTSA  Center  for  Innovation  and 
Technology  Entrepreneurship  (CITE)  competition  and  3rd  place  in  the  ECE  Technology  Symposium. 
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1 .  List  of  Appendixes,  Illustrations  and  Tables  (if  applicable) 


Figure  1  Illustration  of  the  proposed  human-UAV  cooperative  decision  system,  where  UAVs  are  controlled 
actively  (through  a  thought-based  commend)  and  passively  (through  sensing  the  cognitive  states)  by 
soldiers  ’  thoughts  detected  by  EEG  sensors  placed  in  their  helmet. 


Figure  2  The  acquired  Active  Two  256-channel  Biosemi  systems  and  the  EEG  lab  established  at  UTSA 


Figure  3  Block  diagram  of  the  proposed  SSVEP -based  brain-controlled  UAV  system. 

Figure  4  Illustration  in  the  SSVEP  BCI  system  that  implements  6  commends.  (A)  the  arrangement  of  the 
flickering  and  the  frequencies.  (B)  Illustration  of  a  participant  looking  at  flickering  visual  stimulus  in  the 
real  system. 

Figure  5  The  locations  of  the  12  electrodes  used  in  this  system. 

Figure  6  Power  spectrum  for  a  lOHz  (left)  and  I5Hz  (right)  trial.  Different  curves  correspond  to  different 
channels.  Significant  peaks  at  lOHz  (left)  and  I5Hz  (right)  and  their  respective  harmonics  can  be  clearly 
seen. 

Figure  7  Average  error  rates  of  the  experiment  involving  40  participants,  each  conducted  3  trials. 

Figure  8  Screenshots  of  real  demonstration  of  brain-control  quad-copter  flight  missions. 

Figure  9  .  A  The  general  DSN  architecture  with  three  stacking  modules  (dotted  blue  boxes).  Note  input 
(V)  and  output  (Yn-i)  are  concatenated  to  form  the  input  the  2"'^  and  third  module.  B.  The  detailed 
architecture  of  a  DSN  module,  which  is  a  simplified  MLP  that  includes  a  single  layer  RBM  followed  by  a 
linear  classification  layer.  C.  The  detailed  architecture  of  the  proposed  DSNtf  module,  which  includes  a 
set  of  weights  (with  superscript  ‘A”)  trained  from  other  subjects  and  another  set  (with  superscript  “O”) 
trained  specifically  for  the  SOI. 


2.  Statement  of  the  problem  studied 


One  key  objective  of  the  project  in  this  reporting  period  is  to  acquire  high  performance  EEG 
Biosemi  systems.  In  addition  to  our  effort  in  acquiring  the  EEG  systems,  we  have  also  been 
working  on  two  proposed  projects  in  the  area  of  brain  computer  interface  (BCI). 

2.1  Acquisition  of  high  performance  EEG  systems 

The  goal  of  this  project  is  to  acquire  high  performance  Electroencephalogram  (EEG)  systems 
that  enable  real-time  measurement  of  human  brain  activities  at  high  spatial-temporal  resolution 
in  both  laboratory  and  real-life  environments.  The  specific  aims  are  to  enable  1)  multi-subject 
capability;  2)  real-time  measurement  of  brain  wave  with  highest  possible  resolution  at  a 
laboratory  environment;  3)  real-time  measurement  of  brain  activity  in  real  life  environments.  In 
addition,  we  also  intend  to  acquire  additional  EEG  devices  suitable  for  classroom  and  research- 
based  educational  purpose.  We  propose  to  acquire  1)  the  most  advanced  Biosemi  Active  Two 
EEG  device  to  enable  real-time  measurement  of  brain  wave  with  highest  possible  resolution  at  a 
laboratory  environment,  and  2)  dry  electrode,  wearable  wireless  EEG  caps  to  measure  brain 
activities  in  real  life  environment  and  for  educational  use.  For  this  reporting  period,  we  focused 
on  the  acquisition  of  Biosemi  Active  Two  EEG  devices. 

2.2  Controlling  a  Simulated  Quad-copter  using  a  SSVEP-based  Brain-Computer  Interface 


Our  long  term  goal  is  to  develop  and  demonstrate  core  technologies  to  control  cooperative  small 
unmanned  aerial  vehicles  (SUAVs)  to  complete  simple  missions  using  brain  waves  of  a  soldier 
(Fig.  1).  There  has  been  substantial  amount  of  technical  advances  made  over  the  past  two 
decades  in  the  area  of  autonomous  unmanned  vehicles.  The  miniaturization  of  computational, 
sensory,  and  communicational  components  have  accelerated  the  pace  of  the  recent 
advancement.  Most  of  previous  works  dealt  with  a  variety  of  user  interface  issues  for  control 
stations.  What  has  not  been  studied  in  depth  is  ways  for  soldiers  to  interact  directly  with  SUAVs. 
Flowever,  successful  SUAV  mission  executions  rely  heavily  on  the  ability  to  sense  critical 
information  of  a  mission  situation  from  different  complimentary  sensor  resources.  While  these 
sensors  have  proven  to  provide  information  that  can  lead  to  improved  decision  making,  soldiers’ 
comprehension  of  complex  environments,  including  the  collective  sensor  information,  and  their 
cognitive  decisions  are  often  the  most  valuable,  enabling  effective  and  efficient  execution  of 
complex  SUAV  missions.  Integrating  soldiers  into  the  cooperative  decision  making  of  SUAVs  is 
imperative.  Furthermore,  due  to  pervasive  threats  from  enemies  that  employ  unconventional 
tactics,  it  would  be  highly  desirable  to  have  an  integrated,  human-SUAV,  sociotechnical  system, 
where  SUAVs  possess  the  ability  to  sense  the  potential  risks  from  soldiers’  high  level  cognitive 
commands  and  act  accordingly  to  avert  the  risks. 

The  ultimate  goal  of  this  project  is  to  design  a  BCI  system  to  control  a  group  of  SUAVs  to  perform 
a  simple  cooperative  mission,  such  as  a  simple  search  mission  and  an  Intelligence  Collection, 
Surveillance,  and  Reconnaissance  (ISR)  mission  by  only  using  brain  waves  of  a  human.  The 
goal  for  this  reporting  period  is  to  design  a  BCI  system  based  on  the  Steady-State  visually  Evoked 
Potentials  (SSVEP)  method  to  interface  with  a  Quad-copter  simulator  to  control  the  takeoff  (which 
also  hovers)  and  landing  using  brain  wave. 

2.3  A  deep  neural-network  system  for  cross-subject  and  cross-experiment  prediction  of 
image  rapid  serial  visual  presentation  events 


We  propose  to  study  in  the  project  a  particular  BCI  system  that  uses  brain  signals  to  detect  rare 
target  images  within  a  large  collection  of  non-target  images.  Such  BCI-based  systems  can 
significantly  increase  the  speed  of  target  image  identification  in  large  image  databases  over 
manual  procedures.  Data  collection  for  training  such  BCI  systems  is  commonly  carried  out  under 
a  rapid  serial  visual  presentation  (RSVP)  paradigm,  where  test  subjects  are  asked  to  identify  a 
target  image  from  a  continuous  burst  of  image  clips  presented  at  a  high  rate.  The  EEG  recordings 
or  their  power  spectrums  within  300-1 000ms  immediately  following  the  onset  of  target  and  non¬ 
target  images  are  collected  and  a  classifier  capable  of  predicting  the  presence  of  target  images 
based  on  EEG  responses  is  trained  using  this  data. 

The  prediction  of  target  images  using  the  trained  classifier  can  be  categorized  as  a  within-subject, 
cross-subject,  or  cross-experiment  task,  depending  upon,  if  the  prediction  data  come  from  the 
same  subject,  different  subjects,  or  different  RSVP  experiment  as  the  training  data  source, 
respectively.  So  far,  a  number  of  classifiers  including  logistic  regression,  linear  discriminant 
analysis  (LDA),  support  vector  machines  (SVM)  and  hierarchical  discriminant  component  analysis 
(HDCA)  have  been  proposed  in  the  literature.  Although  some  of  these  classifiers  have  been 
shown  to  achieve  high  accuracy  for  within-subject  prediction,  all  of  them  suffer  significant 
performance  degradation  when  applied  for  a  cross-subject  or  cross-experiment  task.  This 
degradation  can  be  largely  attributed  to  the  increased  differences  in  brain  activities  across 
subjects  and  experiments.  As  demonstrated  in  the  literature,  the  target  event  related  potential 
(ERP)  can  change  in  both  magnitude  and  timing  with  subject  and  experiment.  To  improve  the 
prediction  performance,  new  methods  that  can  extract  robust  EEG  features  across  subjects  and 
experiments  and  can  adapt  to  the  changes  of  subjects  and  experiments  need  to  be  developed. 
Moreover,  it  would  be  of  significant  practical  interest  to  develop  a  prediction  system  that  can 
integrate  existing  data  from  other  subjects  with  that  from  the  subject  of  interest  to  improve  the 
prediction  performance.  However,  given  the  potential  differences  in  brain  activities  across 
subjects  and  experiments,  it  is  unclear  if  and  how  data  from  other  subjects/experiments  can  be 
transferred  to  improve  the  target  prediction.  In  this  project,  we  investigated  deep  learning  (DL) 
classifiers  for  cross-subject  and  cross-experiment  prediction  of  RSVP  target  events  and  we  also 
proposed  a  novel  deep  transfer  learning  algorithms  that  can  integrate  data  from  other  subjects  for 
improved  target  prediction.  Deep  learning  is  a  new  family  of  learning  methods  with  a  multiple¬ 
layered  architecture  that  have  been  shown  to  offer  superb  representation  of  complex  data.  DL 
has  gained  great  interest  in  recent  years  due  to  its  ability  to  outperform  other  classification 
methods  in  several  machine  learning  competitions  and  in  a  variety  of  applications  including  image 
classification  and  speech  recognition.  Its  application  in  EEG-based  BCI  data  analysis  is  still  at  a 
very  early  stage.  We  have  previously  shown  on  a  small  dataset  that  deep  learning  algorithms 
achieved  about  5%  performance  gain  over  other  exiting  algorithms  for  within-subject  RSVP  target 
prediction.  However,  more  studies  need  to  be  conducted  to  confirm  the  improvement  of  DL 
classifier  for  within-subject  prediction  based  on  relative  small  training  data  because  the  superior 
performance  of  DL  in  other  applications  is  almost  always  achieved  as  a  result  of  enormous  training 
data  (e.g.  millions  of  training  images).  Furthermore,  if  and  how  DL  can  improve  cross-subject  or 
cross-experiment  prediction  based  on  limited  training  samples  has  not  been  explored. 


3.  Summary  of  the  most  important  results 


We  first  summarize  in  the  following  important  results  for  each  of  the  3  tasks  stated  in  section 
“Statement  of  the  problem  studied”.  We  will  then  discuss  the  outcome  of  outreach  and  educational 
efforts. 


3.1  Acquisition  of  high  performance  EEG  systems 


We  have  purchased  two  (2)  256-channel  Biosemi  Active  Two  EEG  devices.  This  biopotential 
measurement  system  has  256  active  electrodes  and  24-bit  resolution.  It  is  the  most  advanced 
Biosemi  EEG  system  that  offers  to  measure  brain  activity  at  the  highest  possible  spatial  and 
temporal  resolution.  Each  system  includes  an  ActiveTwo  Base  System  that  includes  A/D  Box,  2 
battery  Units,  battery  Charger,  USB  2.0  Interface,  Interface  Cable,  and  acquisition  Software,  an 
amplifier/converter  module,  an  auxiliary  8-channel  input  box,  256  sensors,  and  3  head  cap  with 
different  size.  In  addition,  we  have  acquired  one  64-channel,  one  32-channel,  and  two  16-channel 
Cognionics  dry  sensor  headsets.  We  have  also  set  up  a  110  ft^  EEG  room  for  storing  the  EEG 
systems  and  conducting  experiments  (Fig.  2). 


3.2  Controlling  a  Simulated  Quad-copter  using  a  SSVEP-based  Brain-Computer  Interface 

The  main  objective  of  this  project  is  to  control  a  simple  Unmanned  Aerial  Vehicle  (UAV)  using  the 
brain  activity,  specifically  using  a  Brain-Computer  Interface  (BCI)  system.  The  overall  block 
diagram  of  the  designed  system  can  be  found  in  Fig.  3.  The  main  BCI  element  is  a  Steady-State 
Evoked  Potentials  (SSVEP)  system  that  functions  to  induce  a  specific  pattern  in  brain  signals 
associated  with  a  particular  command  (e.g.  turning  left).  The  command  associated  signals  are 
detected  by  the  Canonical  Correlation  Analysis  (CCA)  algorithm  and  sent  to  the  control  unit  or 
Robot  Operating  System  (ROS)  of  a  quad-copter  (the  UAV),  which  will  execute  the  specific 
command. 


3.2.1  Introduction  to  SSVEP 

Several  approaches  have  been  developed  for  BCI  and  they  make  it  possible  to  control  an 
electronic  device  using  the  subject's  neural  activity.  The  approach  chosen  for  this  project  is  called 
Steady-State  Evoked  Potentials  (SSVEP).  In  brief,  the  main  principle  behind  this  approach  is  that 
when  the  subject  observes  a  visual  stimulus  that  flickers  at  a  certain  frequency  f,  the  neural  activity 
generated  in  the  visual  cortex  registers  (among  other  activities)  a  sinusoidal-like  signal  whose 
frequency  is  also  f.  This  fact  makes  it  possible  to  design  a  BCI  system  that  codes  stimulus 
frequencies  as  commands  to  be  sent  to  the  output  device.  By  accurately  recognizing  the 
frequency  of  the  visual  stimulus  the  subject  is  looking  at,  the  coded  command  can  be  detected. 


SSVEP  belongs  to  the  REACTIVE  type  of  BCI,  which  means  the  subject  is  not  concentrated  in 
giving  direct  orders  with  his  mind.  The  design  of  an  SSVEP  experiment  is  usually  quick,  simple 
and  inexpensive.  SSVEP  is  a  very  active  research  topic  within  the  BCI  field  and  several  detections 


algorithms  have  been  tested  and  diverse  kind  of  applications  have  been  built.  Important  factors 
that  need  to  be  carefully  considered  when  designing  a  SSVEP  system  are  listed  in  the  following: 

•  The  visual  stimulus:  shape,  size,  brightness,  color,  etc.  are  factors  that  influence  the 
SSVEP  performance;  using  flickering  images  as  source  of  stimulus  has  been  documented. 

•  Focal  distance,  concentration  level,  fatigue  and  other  mental  and  physical  states  also 
affect  the  SSVEP  performance. 

•  The  visual  stimulus  are  usually  presented  using  arrays  of  LEDs,  TVs  or  computer  monitor. 
When  using  TV  or  computer  monitors,  the  refresh  rate  of  the  screen  has  to  be  taken  in 
consideration  to  correctly  reproduce  the  flickering  frequencies. 

•  Frequencies  ranging  as  low  as  6Flz  to  high  as  90Flz  have  been  successfully  used  for 
SSVEP. 

•  Minimum  training  is  required  for  an  SSVEP  experiment. 

3.2.2  The  proposed  SSVEP  experiment 

The  objective  of  the  experiment  is  to  capture  the  sine-like  signal  using  EEG,  which  is  elicited  when 
the  subject  watches  the  SSVEP  visual  stimulus.  An  illustration  of  our  SSEVP  experiment  is  shown 
in  Fig.  4.  The  detailed  design  of  the  experiment  can  be  summarized  as  follows: 

•  A  24  inches  computer  monitor  is  placed  in  front  of  the  subject,  approximately  30  -  50  cm 
focal  distance.  The  visual  stimulus  corresponding  to  two  flickering  squares  are  located  on 
the  left  and  right  borders  of  the  screen  at  a  central  high.  Each  square  is  approximately  5 
cm  length  per  side. 

•  Participant  is  wearing  an  EEG  cap  with  12  electrodes  (Fig.  5).  Several  different 
arrangements  of  sensors  on  the  scalp  have  been  used  in  the  literature.  In  our  case,  the 
12  channels  are  spread  throughout  several  locations  around  the  scalp  in  a  symmetric 
fashion,  emphasizing  particularly  the  area  of  visual  cortex.  The  sampling  rate  of  the 
system  is  512Flz. 

•  Visual  stimulus  has  six  flickering  with  O.OFIz,  7.5Flz,  8.57Flz,  lOFIz,  10.9Flz  and  13.3  FIz 
(Fig.  4-(A)). 

•  Participant  starts  in  neutral  position,  looking  at  the  fixation  mark.  An  arrow  indicator 
displays  for  7  seconds,  directing  participant  to  a  specific  flicker.  After  7  seconds,  the  arrow 
disappears  and  the  fixation  mark  appears  for  7  seconds.  This  arrow  rotates  clockwise  with 
the  7  second  intervals.  This  rotation  occurs  3  times  during  a  trial  for  a  total  of  18  views  of 
the  various  flickers.  After  viewing  18  flickers  the  trial  finishes  and  the  flickers  disappears, 
participant  has  40  seconds  to  rest  before  the  next  cycle  begins 

•  Presentation  program  was  implemented  in  MATLAB  using  the  Psychophysics  Toolbox 
(PTB)  for  the  low-level  communications  with  the  computer  monitor  through  OpenGL. 

•  The  lab-streaming  layer  (LSL)  library  was  used  to  stream  the  EEG  recordings  from 
BioSemi  hardware  and  combined  with  the  SSVEP  presentation  data  (event  labels,  timing, 
etc.). 


3.2.3  EEG  acquisition  and  preprocessing 

EEG  signals  were  recorded  using  a  Biosemi  Active  Two  System.  As  mentioned  before,  12 
electrodes  were  used  on  the  standard  adult-size  256  electrodes  cap.  The  SSVEP  presentation 


program  and  the  EEG  recording  were  performed  on  the  same  computer.  For  the  experiment,  EEG 
data  were  recorded  using  the  built-in  Labview-based  Biosemi  acquisition  software. 

The  data  pre-processing  was  conducted  using  MATLAB  and  the  EEGLAB  toolbox.  After  signal 
was  correctly  loaded  and  trimmed  (initial  seconds  of  PTB  initialization),  the  following  operations 
were  carried  out  for  pre-processing: 

•  Downsampling  to  128  Hz 

•  FIR  band-pass  filter  with  frequencies  1  Hz  to  60Hz 

•  Averaged  re-referencing 

•  Baseline  removal 


After  these  steps  of  preprocessing,  the  power  spectrum  was  generated  to  verify  if  there  was 
frequency  spikes  at  the  stimulus  frequency  and  their  harmonics  as  suggested  by  the  SSVEP 
theory.  An  example  of  a  1 0Hz  and  1 5Hz  frequency  response  for  a  single  trial  can  be  seen  in  Fig. 
6. 


3.2.4  Detection  of  SSVEP  frequencies  with  Canonical  Correlation  Analysis 

We  investigated  the  Canonical  Correlation  Analysis  (CCA)  algorithms  for  automatic  detection  of 
the  SSVEP  frequencies  and  evaluated  its  performance  for  our  experiment.  In  a  nutshell,  the  CCA 
algorithm  performs  correlations  between  a  sinusoidal  signal  and  its  harmonics  with  the  predefined 
frequency  (e.g.  10Hz  or  15Hz)  and  EEG  recording  from  different  channels  and  then  finds  a  linear 
transformation  that  maximizes  the  combination  of  the  correlations  or  the  CCA  correlation.  The 
frequency  that  associated  with  the  largest  CCA  correlation  is  detected  as  the  stimulus  frequency. 


3.2.5  Investigate  the  performance  of  the  SSVEP  system. 


To  evaluate  the  performance  of  the  designed  SSVEP  system,  we  conducted  experiments  of  40 
subjects,  each  performed  3  trials  that  last  for  about  1 5  seconds.  Specifically,  the  participant  was 
asked  to  sit  still  in  trial  1  and  2  but  allowed  to  talk  and  move  in  trial  3.  Data  were  spitted  into  a  3- 
second  length  epochs,  where  each  epoch  contains  only  a  single  stimulus.  A  total  of  192  epochs 
were  obtained  from  all  the  recorded  data  for  each  participant  in  each  trial.  CCA  was  applied  to 
each  epoch  and  the  detection  frequency  was  compared  with  the  true  frequency  of  the  flickering. 
An  error  is  record  if  the  predicted  and  the  true  frequencies  did  not  agree.  We  obtain  an  average 
error  rate  of  0.0996,  0.1466  and  0.2936  for  trial  1, 2  and  3,  respective  (Fig  7).  We  observed  that 
the  average  error  rate  is  low  in  trial  1  &2  but  increases  significantly  in  trial  3.  We  performed  T-test 
to  test  the  differences  between  the  trials.  The  test  determines  that  there  is  no  significant 
differences  between  the  error  rate  of  trial  1  &2  (p<0.05)  but  shows  a  significant  difference  between 
the  error  rates  between  trail  1  &3  or  2  &  3.  Since  the  difference  between  trial  1  (or  2)  &  3  is  the 
allowed  movement  of  the  participant,  we  conclude  that  movement  during  SSVEP  experiment 
significantly  impairs  the  detection  performance. 


3.2.6  Execution  of  the  brain-controlled  quadcopter  flight  operations 

As  the  final  step,  we  connected  the  SSVEP  BCI  system  with  the  real  quadcopter  and  performed 
two  types  of  flight  mission.  The  first  mission  involves  one  quadcopter  and  six  low  level  commends 
are  executed,  which  includes  “Move  Forward”,  “Move  Backward”,  “Move  Right”,  “Move  Left”, 
“Move  Up”,  and  “Move  Down”.  Using  these  commends,  we  successfully  demonstrated  the  direct 
execution  of  these  commends  and  the  square  formation.  The  second  mission  includes  controlling 
two  drones  to  execution  high  level  commands  including  “Scatter”  (two  drones  move  away  from 
each  other)  and  “Gather”  (two  drones  move  closer  to  each  other),  which  we  were  able  to 
successfully  implemented  as  well. 


3.3  Deep  learning  methods  for  cross-subject  and  cross-experiment  prediction  of  image 
rapid  serial  visual  presentation  events 

We  investigated  the  performance  of  Deep  Stacking  Network  (DSN)  and  deep  neural  network 
(DNN),  two  most  popular  deep  learning  algorithms,  for  cross-subject  and  cross-experiment 
prediction  of  RSVP  targets.  In  addition,  we  also  proposed  a  new  deep  transfer  learning  algorithm, 
where  we  augment  the  training  data  of  a  subject  of  interest  (SOI)  with  data  from  other  subjects. 


3.3.1  A  new  transfer  learning  deep  stacking  network  algorithm 

The  goal  of  transfer  learning  is  to  augment  the  training  data  of  a  subject  of  interest  (SOI)  with 
data  from  other  subjects  from  either  the  same  of  different  experiment.  However,  simply 
augmenting  data  from  other  subject  might  not  help  improve  the  performance  for  SOI.  Especially, 
the  training  data  from  other  subject  can  be  of  much  larger  size  than  that  from  SOI.  In  this  case, 
the  discriminant  features  of  SOI  can  be  significantly  skewed  by  virtue  of  the  enormous  data  size 
from  other  subjects.  This  could  be  beneficial  when  data  from  other  subject  carries  more 
discriminant  information  but  it  could  equally  be  detrimental  when  target  event  related  brain 
activities  from  other  subjects  are  not  as  discriminant.  Therefore,  it  is  desirable  to  develop  an 
algorithm  that  can  transfer  only  the  useful,  discriminant  information  of  other  subject  to  SOI.  We 
propose  a  novel  DSN  architecture  for  transfer  learning  or  DSNtf  (Fig.  9  ).  The  uniqueness  of 
DSNtf  is  that  it  is  essentially  an  augmentation  of  a  SOI  DSN  with  a  cross-subject  DSN.  To  train 
this  DSNtf,  we  first  train  a  cross-subject  DSN  suing  data  from  other  subjects.  Then,  the  SOI 
DSN  is  trained  using  only  data  from  SOI.  However,  before  training,  the  hidden  units  in  each 
module  are  expanded  by  including  those  from  the  trained  cross-subject  DSN,  where  the 
corresponding  RBM  weights  for  the  cross-subject  hidden  units  are  also  retained.  The  training 
thus  involves  learning  the  weights  for  the  SOI  hidden  units  as  well  as  updating  the  weights  for 
cross-subject  hidden  units  using  the  subject-specific  data.  Because  the  training  of  each  DSNtf 
module  should  maximize  the  classification  performance,  weight  updating  for  cross-subject 
hidden  units  functions  to  transfer  the  discriminant  information  learned  from  other  subjects  about 
the  target  event  to  the  individualized  DSN  designed  for  the  subject  of  interest.  As  a  result,  this 
new  architecture  integrates  subject-specific  knowledge  and  cross-subjects  knowledge  about  the 
target  event  to  produce  a  prediction  that  not  only  emphasizes  individual  characteristics  in  brain 
activity  but  also  reflects  the  subject-wide  consensus  that  defines  the  global  brain  activity 


underlying  the  target  event.  Because  the  cross-subject  hidden  units  bring  the  global,  potentially 
more  robust  features  into  the  classification,  this  new  architecture  is  likely  immune  to  data  noise 
in  outlier  samples,  thus  making  the  individualized  prediction  more  robust.  At  the  same  time, 
when  cross-subject  hidden  units  are  less  discriminant,  the  updating  process  should  function  to 
discount  these  cross-subject  hidden  units,  thus  maintaining  a  performance  close  to  that  of  the 
within-subject. 

3.3.2  DL  algorithms  showed  significant  improvement  over  existing  algorithms. 

We  evaluated  the  performance  of  DSN  and  DNN  for  within-subject  target  classification  for  two 
RSVP  datasets  ,  the  static  motion  and  the  Cognitive  Technology  Threat  Warning  System 
(CT2WS).  The  static  motion  experiment  include  presentation  of  images  of  enemy 
soldiers/combatants  (target)  versus  the  background  image  (non-target).  In  the  CT2WS 
experiment,  target  images  include  moving  people  and  vehicle  animations,  whereas  the  non¬ 
targets  are  other  types  of  animation.  For  both  experiments,  the  images  were  presented  at  2Hz 
(one  image  presented  every  500ms)  and  brain  signals  were  recorded  with  64-channel  Biosemi 
EEG  systems  at  a  sampling  rate  of  512  Hz.  There  were  a  total  of  16  and  15  subjects  in  the  static 
motion  and  CT2WS  experiment,  respectively.  For  both  experiments,  DNN  and  DSN  both  had 
very  similar  Area  Under  the  ROC,  or  Az  performance  and  DNN  reported  an  Az  score  of  94.5%  for 
the  static  motion  RSVP  experiment.  When  compared  with  the  state-of-the-art  HDCA  method,  the 
two  DL  algorithms  reported  >  6%  improvements  in  Az  scores  for  both  experiments.  Taken 
together,  this  result  demonstrated  good  performance  and  a  clear  improvement  achieved  by  DL 
algorithms  for  within-subject  target  prediction. 


3.3.3  DL  classifiers  are  more  robust  for  cross-subject  and  cross-experiment  predictions. 

We  investigated  how  robust  DL  algorithms  when  used  to  predict  the  target  event  for  a  test 
subject  different  from  those  for  training.  We  first  evaluated  their  cross-subjects  performance, 
where  the  test  subject  and  training  subjects  are  from  the  same  experiment.  As  expected,  both 
HDCA  and  LDA  deteriorated  significantly  relative  to  their  within-subject  performance,  where  the 
Az  scores  of  HDCA  dropped  for  more  than  1 0%  from  0.87  to  0.74  for  the  static  motion  and  0.8  to 
0.7  for  the  CT2WS.  In  contrast,  both  DNN  and  DSN  demonstrated  a  much  robust  performance, 
suffered  only  4.7%  and  2.1%  performance  degradation  for  the  static  motion  and  0.8  to  0.7  for  the 
CT2WS,  respectively. 

We  then  examined  if  this  more  robust  performance  can  still  be  observed  for  cross-experiment 
prediction,  which  is  a  more  difficult  task  than  cross-subject  prediction  because  the  test  subject  is 
from  an  experiment  different  from  the  training  subject.  For  this  test,  we  trained  the  classifiers 
using  all  subject  data  from  the  static  motion  RSVP  experiment  and  then  evaluated  their 
performance  for  each  subjects  from  the  CT2WS  experiment.  As  expected,  the  performance  of  all 
tested  classifiers  including  DNN  and  DSN  degraded  more  than  cross-subject  performance. 
However,  the  two  DL  algorithms  registered  a  less  degradation  in  performance  than  either  HDCA 
or  LDA  (-12%  for  DL  from  0.86  to  0.76  vs.  -16%  for  HDCA  from  0.8  to  0.67  and  28%  for  LDA 
from  0.74  to  0.58).  Taken  together,  the  results  indicated  that  DL  algorithms  are  more  robust  for 
both  cross-subject  and  cross-experiment  predictions. 


3.3.4  The  proposed  transfer  learning  DSN  improves  over  the  within-subject  prediction 
performance 


3.3.5 


We  investigated  if  the  proposed  transfer  learning  schemes  can  improve  the  within-subject 
performance  by  integrating  additional  data  from  other  subjects  either  from  the  same  experiment 
or  different  experiment.  For  the  within-experiment  transfer  learning,  similar  procedure  as  the 
within-subject  test  was  followed  except  that  the  within-subject  training  data  is  augmented  also  by 
data  from  other  subjects  of  the  same  experiment.  Surprisingly,  directly  augmenting  the  training 
data  from  other  subjects  appeared  to  harm  rather  than  improve  the  within-subject  performance. 
This  result  suggests  that  there  exist  differences  in  the  learned  discriminant  EEG  features  for 
different  subjects  and  many  discriminant  EEG  features  for  other  subjects  would  not  be 
discriminant  for  the  subject  of  interest.  As  a  result,  including  data  from  other  subjects  might  add 
more  non-discriminant  features  and  subsequently  degrade  the  performance.  However,  the  fact 
that  DNN  with  data  augmentation  (DNNda)  and  DSN  with  data  augmentation  (DSNda)  only 
suffered  a  slight  drop  in  performance  suggests  once  again  that  the  discriminant  features  learnt  by 
the  DL  algorithms  are  more  robust  cross  subjects.  In  contract,  the  performance  of  HDCA  and  LDA 
deteriorated  much  closer  to  their  corresponding  cross-subject  performance.  This  observation 
could  suggest  that  the  discriminant  EEG  features  learnt  by  HDCA  and  LDA  cross  subjects  are 
potentially  very  different.  The  learnt  discriminant  features  became  similar  to  those  by  cross¬ 
subject  training,  hence  the  performance,  because  after  data  augmentation  there  were  much  more 
data  samples  from  cross  subjects  than  subject  of  interest. 

On  the  contrary,  we  noticed  that  the  proposed  transfer  learning  DSN  did  achieved  improvement 
over  the  within-  subject  DSN.  This  result  substantiated  our  original  motivation  for  proposing  the 
transfer  learning  architecture.  Indeed,  this  TF  architecture  enables  to  transfer  these  cross¬ 
subjects  discriminant  features  that  are  only  beneficial  to  the  subject  of  interest.  The  result  that 
DSNtf  achieved  almost  the  same  performance  as  within-subject  DSN  for  the  CT2WS  experiment 
also  demonstrated  that  when  the  cross-subjects  data  appear  very  different,  this  architecture 
seems  to  allow  potentially  no  features  transferred. 

We  further  validated  this  result  about  transfer  learning  by  augmenting  data  of  subjects  from 
different  experiments.  HDCA  and  LDA  suffered  more  significant  deterioration  because  of  the 
impact  of  additional  cross-experimental  differences.  As  expected,  DSNda  and  DNNda  only  have 
small  drops  in  performance.  Overall,  DSNtf  still  reported  the  best  performance  which  is  only  1% 
lower  than  that  of  within-subject  DSN.  This  result  confirmed  that  the  conceived  advantages  of 
the  proposed  transfer  learning  DSN. 

3.4  Education  and  outreach  activities 

Overall,  we  have  involved  2  PhD  students,  one  MS  student,  and  2  undergraduate  students  in 
conducting  research  that  are  related  to  this  project. 

We  hosted  Kristine  Diaz,  a  female,  Hispanic  undergraduate  student  from  the  Alamo  College  to 
implement  a  SSVEP  based  BCI  system  from  July-Aug,  2014.  Kristine  Diaz  was  selected  through 
the  Alamo  College  Summer  Research  Program,  a  National  Science  Foundation  funded  program. 
Ms.  Diaz  has  participated  in  the  SSVEP  experiments  both  as  a  test  subject  and  in  helping  out  with 
the  experiments.  She  has  also  involved  in  data  collection,  preprocessing,  and  analysis.  She  has 
reported  her  research  experience  and  results  from  this  project  in  a  poster  presented  the  final 
research  conference  of  the  program. 

We  have  also  recruited  Jonathan  David  Upchurch  and  Garrett  Hall,  two  UTSA  electrical 
engineering  undergraduate  students  and  Air  Force  veterans,  to  work  on  the  brain-controlled  UAV 
project.  Both  of  them  have  been  helping  conducting  SSVEP  experiments  and  research  the  dry 
sensor  EEG  systems. 


We  have  also  mentored  an  EE  senior  design  team  to  implement  a  BCI  system  that  monitors 
specific  areas  in  the  frontal  lobe  that  deal  with  concentration  and  focus.  The  senior  design  project 
won  the  2^^  place  in  the  UTSA  Center  for  Innovation  and  Technology  Entrepreneurship  (CITE) 
competition  and  3'^'*  place  in  the  ECE  Technology  Symposium. 


Figures 


Figure  1  Illustration  of  the  proposed  human-UAV  cooperative  decision  system,  where  UAVs  are 
controlled  actively  (through  a  thought-based  commend)  and  passively  (through  sensing  the  cognitive 
states)  by  soldiers  ’  thoughts  detected  by  EEG  sensors  placed  in  their  helmet. 


Fieure  2  The  acauired  Active  Two  256-channel  Biosemi  .svstems  and  the  EEG  lab  established  at 
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Figure  3  Block  diagram  of  the  proposed  SSVEP -based  brain-controlled  UAV  system. 


m 

□ 

10.9Hz  7.5Hz 

13.3Hz 

a 

10Hz 

^row  1  Fixation 

m 

6Hz 

indicator  nfiark 

Q  8.57Hz 

(A) 


(B) 


Figure  4  Illustration  in  the  SSVEP  BCI  system  that  implements  6  commends.  (A)  the  arrangement  of 
the  flickering  and  the  frequencies.  (B)  Illustration  of  a  participant  looking  at  flickering  visual  stimulus 
in  the  real  system. 


Figure  5  The  locations  of  the  12  electrodes  used  in  this  system. 


Figure  6  Power  spectrum  for  a  lOHz  (left)  and  15Hz  (right)  trial.  Different  curves  correspond  to 
different  channels.  Significant  peaks  at  lOHz  (left)  and  15Hz  ( right)  and  their  respective  harmonics  can 
be  clearly  seen. 
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BCI  commands  performance  per  Trial  1,  4.5s,  all  frequencies  (192  values) 
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BCI  Command 

BCI  commands  performance  per  Trial  2,  4.5s,  all  frequencies  (192  values) 
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BCI  commands  performance  per  Trial  3,  4.5s,  all  frequencies  (192  values) 
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Figure  7  Average  error  rates  of  the  experiment  involving  40  participants,  each  conducted  3  trials. 


Figure  8  Screenshots  of  real  demonstration  of  brain-control  quad-copter  flight  missions. 
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A.  3-Layer  DSN  Model  B.  h*  DSN  Module  C.  DSN^  Module 


Figure  9  .  A  The  general  DSN  architecture  with  three  stacking  modules  (dotted  blue  boxes).  Note  input 
(V)  and  output  (Yn-i)  are  concatenated  to  form  the  input  the  2"^^  and  third  module.  B.  The  detailed 
architecture  of  a  DSN  module,  which  is  a  simplified  MLP  that  includes  a  single  layer  RBM  followed  by 
a  linear  classification  layer.  C  The  detailed  architecture  of  the  proposed  DSNtf  module,  which 
includes  a  set  of  weights  ( with  superscript  “A  ”)  trained  from  other  subjects  and  another  set  ( with 
superscript  “O  ”)  trained  specifically  for  the  SOI. 


